Abstract:

Computerized adaptive assessment of knowledge has become a prominent testing mode in the educational assessment. It is essentially developed on the basis of two components: (1) an algorithm that selects the items to be administered and (2) a very large pool of items. In the present project, the framework for building a computerized adaptive test is knowledge space theory (KST). One of the most prominent features of KST, compared to psychometric testing (both classical test theory and item response theory), is that no attempt is made to compute a numerical score for representing the knowledge level of a student. Rather, the goal of the assessment is to describe what the student knows and does not. In KST the student's knowledge is represented by the subset of all problems that she is capable of solving in ideal conditions. This subset is the knowledge state. The collection of all knowledge states existing in a population of student is the knowledge structure. One of the basic properties of a knowledge structure is that, among the problems belonging to the knowledge domain, logical or pedagogical relationships can be found. This property is fundamental for the development of an efficient adaptive procedure, because it allows uncovering the individual knowledge state by asking a minimum number of questions.
Among adaptive procedures existing in KST, probabilistic continuous procedures will be considered. In such procedures, the likelihood function on the knowledge states is updated according to the evidence coming from the student’s responses. At each point in time the likelihood captures the current plausibility of the knowledge states. The assessment stops when a large enough portion of the likelihood is concentrated on a single knowledge state. The likelihood function is that of the basic local independence model (BLIM). The BLIM is a probabilistic model in which three types of parameters are defined: the probabilities of the knowledge states in the population, the probability to observe a correct answer by the student, given that the problem does not belong to his knowledge state (lucky guess probability), and the probability to observe a wrong answer, given that the problem belongs to the student’s knowledge state (careless error probability). A very strong assumption in the BLIM is that the carless error and lucky guess parameters are a properties of the items, and do not vary with students. This assumption is too strong and not very realistic.
In the first phase of the research, two extensions of the BLIM, in which the error parameters vary with both items and students, will be developed and compared to one another in three different studies. The aim of the first phase is to develop, test and apply an efficient and accurate adaptive procedure for the assessment of knowledge that overcomes the limitations of current BLIM-based procedures. The second phase of the research will be focused on the construction of pools of items, to be used by the adaptive procedure. Dynamic items can be viewed as abstract templates for generating instances, of the same item, on the fly. This type of items is very useful and powerful, since they potentially allow the automatic generation of infinite instances of the same item. However, special attention must be given to the definition of the rules that generate the dynamic components of a dynamic item: Ideally (without carless errors or lucky guesses), the same student should provide the same (correct or wrong) answer to all instances of the same dynamic item. Despite these critical aspects, there are very few studies that systematically analyze “homogeneity” among the instances of a dynamic item. Simulation and empirical studies will be specifically focused on how the dynamic component specification rules affect instance homogeneity.